Information processing device, information processing method, computer program, and mobile device

ABSTRACT

There is provided an information processing device that processes detection information of an external recognition sensor.The information processing device includes: a recognition unit that performs recognition processing on an object on the basis of a detection signal of the sensor; and a processing unit that performs fusion processing on first data before the recognition by the recognition unit and another data. The information processing device further includes a second recognition unit that performs recognition processing on the object on the basis of a detection signal of a second sensor. The processing unit performs fusion processing on third data before the recognition by the second recognition unit and the first data, fusion processing on fourth data after the recognition by the second recognition unit and the first data, and the like.

TECHNICAL FIELD

The technology disclosed in the present specification relates to aninformation processing device, an information processing method, acomputer program, and a mobile device that process information detectedby a plurality of sensors for mainly recognizing the external world.

BACKGROUND ART

To implement automated driving and advanced driver assistance system(ADAS), it is necessary to detect various objects such as othervehicles, persons, and lanes. Furthermore, it is necessary to detectobjects not only in the daytime in fine weather but also in variousenvironments such as rainy weather and at night. For this reason, manyexternal recognition sensors of different types, such as cameras,millimeter wave radars, and laser radars, are beginning to be installedin vehicles.

Each sensor has its strong and weak points. The recognition performanceof a sensor may deteriorate depending on the type and size of the objectto be detected, the distance to the object, the weather at the time ofdetection, or the like. For example, a vehicle-mounted radar has highdistance accuracy and relative speed accuracy, but low angle accuracy,and does not have an identification function of identifying the type ofobject, or identification accuracy is low. Meanwhile, a camera hasrelatively low distance accuracy and relative speed accuracy, but hasgood angle accuracy and identification accuracy.

Therefore, not only using each sensor alone, but also combining two ormore sensors to take advantage of characteristics of each sensorcontributes to more accurate external recognition. The combination oftwo or more sensors will be hereinafter referred to as “sensor fusion”or “fusion”.

For example, a road traffic monitoring system has been proposed in whicha combination of a plurality of sensors with different detectioncharacteristics is switched and used on the basis of environmentalindicators such as temperature data, rainfall data, visibility data, andilluminance data (see Patent Document 1).

Furthermore, a vehicle traveling control system has been proposed thatprepares a plurality of fusion specifications for the externalenvironment and notifies a driver of a detection area of a sensor whererecognition accuracy decreases due to the external environment and callsattention in the selected fusion specification (see Patent Document 2).

CITATION LIST Patent Document Patent Document 1: Japanese PatentApplication Laid-Open No. 2003-162795 Patent Document 2: Japanese PatentApplication Laid-Open No. 2017-132285 SUMMARY OF THE INVENTION Problemsto be Solved by the Invention

An object of the technology disclosed in the present specification is toprovide an information processing device, an information processingmethod, a computer program, and a mobile device that perform fusionprocessing on a plurality of sensors for mainly recognizing the externalworld.

Solutions to Problems

A first aspect of technology disclosed in the present specification isan information processing device including:

a recognition unit configured to perform recognition processing on anobject on the basis of a detection signal of a sensor; and

a processing unit configured to perform fusion processing on first databefore the recognition by the recognition unit and another data.

The sensor includes, for example, a millimeter wave radar. Then, beforethe recognition, the recognition unit performs processing of each ofdistance detection, speed detection, angle detection of the object, andtracking of the object on the basis of the detection signal of thesensor, and the first data includes at least one of the detectionsignal, a distance detection result of the object, a speed detectionresult, an angle detection result, or a tracking result of the object.Then, the processing unit may perform at least one fusion processing offusion processing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data.

Furthermore, the information processing device according to the firstaspect further includes a second recognition unit that performsrecognition processing on an object on the basis of a detection signalof a second sensor including at least one of a camera or a LiDAR. Then,in a case where the recognition result by the second recognition unit isgood but the recognition result by the recognition unit is not good, theprocessing unit performs fusion processing on the first data.Alternatively, in a case where the recognition result by the secondrecognition unit is not good, the processing unit performs fusionprocessing on the first data.

Furthermore, a second aspect of the technology disclosed in the presentspecification is an information processing method including:

a recognition step of performing recognition processing on an object onthe basis of a detection signal of a sensor; and

a processing step of performing fusion processing on first data beforethe recognition in the recognition step and another data. The processingstep may include performing at least one fusion processing of fusionprocessing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data.

Furthermore, a third aspect of the technology disclosed in the presentspecification is a computer program described in a computer-readableformat for causing a computer to function as:

a recognition unit configured to perform recognition processing on anobject on the basis of a detection signal of a sensor; and

a processing unit configured to perform fusion processing on first databefore the recognition by the recognition unit and another data. Theprocessing unit may perform at least one fusion processing of fusionprocessing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data.

The computer program according to the third aspect defines a computerprogram described in a computer-readable format so as to implementpredetermined processing on a computer. In other words, by installingthe computer program according to the third aspect in the computer, acollaborative action is exhibited on the computer, and similar actionsand effects to the information processing device according to the firstaspect can be obtained.

Furthermore, a fourth aspect of the technology disclosed in the presentspecification is a mobile device including:

a moving means;

a sensor;

a recognition unit configured to perform recognition processing on anobject on the basis of a detection signal of the sensor;

a processing unit configured to perform fusion processing on first databefore the recognition by the recognition unit and another data; and

a control unit configured to control the moving means on the basis of aprocessing result of the processing unit. The processing unit mayperform at least one fusion processing of fusion processing on thirddata before the recognition by the second recognition unit and the firstdata, fusion processing on fourth data after the recognition by thesecond recognition unit and the first data, fusion processing on thefirst data and second data after the recognition by the recognitionunit, or fusion processing on the fourth data and the second data.

Effects of the Invention

The technology disclosed in the present specification can provide aninformation processing device, an information processing method, acomputer program, and a mobile device that perform fusion processing ona plurality of sensors for mainly recognizing the external world.

Note that effects described in the present specification are merelyillustrative, and effects of the present invention is not limited tothese effects. Furthermore, the present invention may produce additionaleffects in addition to the effects described above.

Still another object, feature, and advantage of the technology disclosedin the present specification will be apparent from more detaileddescriptions based on the embodiment as described later and theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a schematic functional configurationexample of a vehicle control system 100.

FIG. 2 is a diagram showing a functional configuration of an informationprocessing device 1000.

FIG. 3 is a diagram illustrating an image captured by a camera.

FIG. 4 is a diagram illustrating a scene detected by a millimeter waveradar.

FIG. 5 is a diagram showing a result of detecting the scene shown inFIG. 4 with the millimeter wave radar.

FIG. 6 is a diagram showing an internal configuration example of a radarrecognition processing unit 1020.

FIG. 7 is a view showing one example of a scene to be recognized.

FIG. 8 is a diagram showing data before recognition processing on thescene shown in FIG. 7.

FIG. 9 is a diagram showing data after recognition processing on thescene shown in FIG. 7.

FIG. 10 is a view showing another example of the scene to be recognized.

FIG. 11 is a diagram showing data before recognition processing on thescene shown in FIG. 10.

FIG. 12 is a diagram showing data after recognition processing on thescene shown in FIG. 10.

FIG. 13 is a diagram showing an example of a fusion processing result(late fusion processing only) by the information processing device 1000.

FIG. 14 is a diagram showing an example of the fusion processing result(including early fusion processing) by the information processing device1000.

FIG. 15 is a diagram showing an example in which results differ betweenlate fusion processing and early fusion processing.

FIG. 16 is a diagram showing an example in which results differ betweenlate fusion processing and early fusion processing.

FIG. 17 is a diagram showing a configuration example of the informationprocessing device 1000 configured to perform early fusion processingadaptively.

FIG. 18 is a flowchart showing a processing procedure for performingtarget recognition in the information processing device 1000 shown inFIG. 17.

FIG. 19 is a diagram showing another configuration example of theinformation processing device 1000 configured to perform early fusionprocessing adaptively.

FIG. 20 is a flowchart showing the processing procedure for performingthe target recognition in the information processing device 1000 shownin FIG. 19.

FIG. 21 is a diagram for describing processing of recognizing an objectthat cannot be recognized by a recognizer 1023 on the basis of RAW dataof a millimeter wave radar 1080.

FIG. 22 is a diagram for describing the processing of recognizing theobject that cannot be recognized by the recognizer 1023 on the basis ofthe RAW data of the millimeter wave radar 1080.

FIG. 23 is a diagram for describing the processing of recognizing theobject that cannot be recognized by the recognizer 1023 on the basis ofthe RAW data of the millimeter wave radar 1080.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment of the technology disclosed in the presentspecification will be described in detail with reference to thedrawings.

FIG. 1 is a block diagram showing a schematic functional configurationexample of a vehicle control system 100, which is one example of amobile body control system to which the present technology can beapplied.

Note that hereinafter, in a case where a vehicle provided with thevehicle control system 100 is distinguished from other vehicles, thevehicle is referred to as own vehicle or self vehicle.

The vehicle control system 100 includes an input unit 101, a dataacquisition unit 102, a communication unit 103, an inside-vehicle device104, an output control unit 105, an output unit 106, a drive-affiliatedcontrol unit 107, a drive-affiliated system 108, a body-affiliatedcontrol unit 109, a body-affiliated system 110, a storage unit 111, andan automated driving control unit 112. The input unit 101, the dataacquisition unit 102, the communication unit 103, the output controlunit 105, the drive-affiliated control unit 107, the body-affiliatedcontrol unit 109, the storage unit 111, and the automated drivingcontrol unit 112 are connected to each other via a communication network121. The communication network 121 includes, for example, avehicle-mounted communication network, a bus, and the like conforming toarbitrary standards such as controller area network (CAN), localinterconnect network (LIN), local area network (LAN), or FlexRay(registered trademark). Note that each unit of the vehicle controlsystem 100 may be directly connected without going through thecommunication network 121.

Note that hereinafter, in a case where each unit of the vehicle controlsystem 100 performs communication via the communication network 121, thedescription of the communication network 121 will be omitted. Forexample, in a case where the input unit 101 and the automated drivingcontrol unit 112 communicate with each other via the communicationnetwork 121, it is simply described that the input unit 101 and theautomated driving control unit 112 communicate with each other.

The input unit 101 includes a device to be used by an occupant to inputvarious data, instructions, and the like. For example, the input unit101 includes an operation device such as a touch panel, a button, amicrophone, a switch, and a lever, and an operation device and the likethat allows input by a method other than manual operation such as voiceand gesture. Furthermore, for example, the input unit 101 may be aremote control device using infrared rays or other radio waves, or maybe an externally connected device including a mobile device, a wearabledevice, or the like that supports the operation of the vehicle controlsystem 100. The input unit 101 generates an input signal on the basis ofdata, instructions, and the like input by an occupant, and supplies theinput signal to each unit of the vehicle control system 100.

The data acquisition unit 102 includes various sensors and the like thatacquire data to be used for processing by the vehicle control system100, and supplies the acquired data to each unit of the vehicle controlsystem 100.

For example, the data acquisition unit 102 includes various sensors fordetecting the state of the own vehicle and the like. Specifically, forexample, the data acquisition unit 102 includes a gyro sensor, anacceleration sensor, an inertial measurement unit (IMU), and sensors andthe like for detecting an accelerator pedal operation amount, a brakepedal operation amount, a steering wheel steering angle, the number ofengine rotations, the number of motor rotations, a wheel rotation speed,or the like.

Furthermore, for example, the data acquisition unit 102 includes varioussensors for detecting information outside the own vehicle. Specifically,for example, the data acquisition unit 102 includes an image capturingdevice such as a time of flight (ToF) camera, a stereo camera, amonocular camera, an infrared camera, and other cameras. Furthermore,for example, the data acquisition unit 102 includes an environmentalsensor for detecting the weather, atmospheric phenomena, or the like,and a surrounding information detection sensor for detecting an objectaround the own vehicle. The environmental sensor includes, for example,a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, andthe like. The surrounding information detection sensor includes, forexample, an ultrasonic wave sensor, a millimeter wave radar, a lightdetection and ranging, laser imaging detection and ranging (LiDAR), asonar, and the like.

Moreover, for example, the data acquisition unit 102 includes varioussensors for detecting the current position of the own vehicle.Specifically, for example, the data acquisition unit 102 includes aglobal navigation satellite system (GNSS) receiver and the like thatreceive a GNSS signal from a GNSS satellite.

Furthermore, for example, the data acquisition unit 102 includes varioussensors for detecting inside-vehicle information. Specifically, forexample, the data acquisition unit 102 includes an image capturingdevice that captures an image of a driver, a biometric sensor thatdetects biometric information on the driver, a microphone that collectsvoice inside the vehicle, and the like. The biometric sensor isprovided, for example, on a seat surface, a steering wheel, or the like,and detects biometric information on an occupant seated on a seat or adriver holding the steering wheel.

The communication unit 103 communicates with the inside-vehicle device104 and various devices, servers, base stations, and the like outsidethe vehicle. The communication unit 103 transmits data supplied fromeach unit of the vehicle control system 100, and supplies the receiveddata to each unit of the vehicle control system 100. Note that thecommunication protocol supported by the communication unit 103 is notparticularly limited, and furthermore, the communication unit 103 cansupport a plurality of types of communication protocols.

For example, the communication unit 103 performs wireless communicationwith the inside-vehicle device 104 by wireless LAN, Bluetooth(registered trademark), near field communication (NFC), wireless USB(WUSB), or the like. Furthermore, for example, the communication unit103 performs wired communication with the inside-vehicle device 104 viaa connection terminal (not shown) (and a cable if necessary) byuniversal serial bus (USB), high-definition multimedia interface (HDMI),mobile high-definition link (MHL), or the like.

Moreover, for example, the communication unit 103 performs communicationwith a device (for example, application server or control server)existing on an external network (for example, the Internet, a cloudnetwork, or a network peculiar to a business operator) via a basestation or access point. Furthermore, for example, the communicationunit 103 performs communication with a terminal existing near the ownvehicle (for example, pedestrian terminal or store terminal, or machinetype communication (MTC) terminal) by using peer to peer (P2P)technology. Moreover, for example, the communication unit 103 performsV2X communication including vehicle-to-vehicle communication,vehicle-to-infrastructure communication, vehicle-to-home communication,vehicle-to-pedestrian communication, and the like. Furthermore, forexample, the communication unit 103 includes a beacon receiving unit,receives radio waves or electromagnetic waves transmitted from awireless station and the like installed on a road, and acquiresinformation including the current position, traffic congestion, trafficregulations, required time, or the like.

The inside-vehicle device 104 includes, for example, a mobile device ora wearable device owned by an occupant, an information device carried inor attached to the own vehicle, a navigation device for searching for aroute to an arbitrary destination, and the like.

The output control unit 105 controls output of various pieces ofinformation to the occupant of the own vehicle or the outside of thevehicle. For example, the output control unit 105 generates an outputsignal including at least one of visual information (for example, imagedata) or auditory information (for example, voice data) and supplies theoutput signal to the output unit 106, thereby controlling the output ofthe visual information and the auditory information from the output unit106. Specifically, for example, the output control unit 105 combinesimage data captured by different image capturing devices of the dataacquisition unit 102 to generate a bird's-eye image, a panoramic image,or the like, and supplies the output signal including the generatedimage to the output unit 106. Furthermore, for example, the outputcontrol unit 105 generates voice data including a warning sound, awarning message, or the like for dangers including collision, scrape,entry into a danger zone, and the like, and supplies the output signalincluding the generated voice data to the output unit 106.

The output unit 106 includes a device that can output visual informationor auditory information to an occupant of the own vehicle or the outsideof the vehicle. For example, the output unit 106 includes a displaydevice, an instrument panel, an audio speaker, a headphone, a wearabledevice including a glasses-type display worn by an occupant and thelike, a projector, a lamp, and the like. In addition to a device havinga regular display, the display device included in the output unit 106may be, for example, a device that displays visual information withinthe field of view of the driver, including a head-up display, atransmissive display, a device having an augmented reality (AR) displayfunction, and the like.

The drive-affiliated control unit 107 generates various control signalsand supplies the control signals to the drive-affiliated system 108,thereby controlling the drive-affiliated system 108. Furthermore, thedrive-affiliated control unit 107 supplies a control signal to each unitother than the drive-affiliated system 108 as necessary, and performsnotification of the control state of the drive-affiliated system 108,and the like.

The drive-affiliated system 108 includes drive-affiliated variousdevices of the own vehicle. For example, the drive-affiliated system 108includes a driving force generation device for generating driving forceincluding an internal combustion engine, driving motor, or the like, adriving force transmission mechanism for transmitting the driving forceto wheels, a steering mechanism that adjusts the steering angle, abraking device that generates braking force, an antilock brake system(ABS), an electronic stability control (ESC), an electric power steeringdevice, and the like.

The body-affiliated control unit 109 generates various control signalsand supplies the control signals to the body-affiliated system 110,thereby controlling the body-affiliated system 110. Furthermore, thebody-affiliated control unit 109 supplies a control signal to each unitother than the body-affiliated system 110 as necessary, and performsnotification of the control state of the body-affiliated system 110, andthe like.

The body-affiliated system 110 includes various body-affiliated devicesequipped in the vehicle body. For example, the body-affiliated system110 includes a keyless entry system, a smart key system, a power windowdevice, a power seat, a steering wheel, an air conditioner, variouslamps (for example, head lamp, reverse lamp, stop lamp, directionindicator lamp, fog lamp, and the like), and the like.

The storage unit 111 includes, for example, a read only memory (ROM), arandom access memory (RAM), a magnetic storage device such as a harddisc drive (HDD), a semiconductor storage device, an optical storagedevice, an optical magnetic storage device, and the like. The storageunit 111 stores various programs, data, and the like to be used by eachunit of the vehicle control system 100. For example, the storage unit111 stores map data including a three-dimensional high-precision mapsuch as a dynamic map, a global map that has precision lower than thehigh-precision map and covers a large area, a local map that includesinformation around the own vehicle, and the like.

The automated driving control unit 112 controls automated drivingincluding autonomous traveling, driving assistance, or the like.Specifically, for example, the automated driving control unit 112performs cooperative control aimed at implementing functions of anadvanced driver assistance system (ADAS) including collision avoidanceor impact mitigation of the own vehicle, follow-up traveling based ondistance between vehicles, traveling while maintaining vehicle speed,collision warning of the own vehicle, lane deviation warning of the ownvehicle, or the like. Furthermore, for example, the automated drivingcontrol unit 112 performs cooperative control aimed at automated drivingand the like in which the vehicle autonomously travels without dependingon the operation of the driver. The automated driving control unit 112includes a detection unit 131, a self-position estimation unit 132, asituation analysis unit 133, a planning unit 134, and an operationcontrol unit 135.

The detection unit 131 detects various types of information necessaryfor controlling automated driving. The detection unit 131 includes anoutside-vehicle information detection unit 141, an inside-vehicleinformation detection unit 142, and a vehicle state detection unit 143.

The outside-vehicle information detection unit 141 performs detectionprocessing on information outside the own vehicle on the basis of dataor signals from each unit of the vehicle control system 100. Forexample, the outside-vehicle information detection unit 141 performsdetection processing, recognition processing, and tracking processing onan object around the own vehicle, and detection processing on thedistance to the object. The object to be detected includes, for example,a vehicle, a person, an obstacle, a structure, a road, a traffic light,a traffic sign, a road marking, and the like. Furthermore, for example,the outside-vehicle information detection unit 141 performs detectionprocessing on the environment around the own vehicle. The surroundingenvironment to be detected includes, for example, weather, temperature,humidity, brightness, road surface condition, and the like. Theoutside-vehicle information detection unit 141 supplies data indicatinga result of the detection processing to the self-position estimationunit 132, a map analysis unit 151, a traffic rule recognition unit 152,and a situation recognition unit 153 of the situation analysis unit 133,an emergency avoidance unit 171 of the operation control unit 135, andthe like.

The inside-vehicle information detection unit 142 performs detectionprocessing on information inside the vehicle on the basis of data orsignals from each unit of the vehicle control system 100. For example,the inside-vehicle information detection unit 142 performs driverauthentication processing and recognition processing, driver statedetection processing, occupant detection processing, inside-vehicleenvironment detection processing, and the like. The driver state to bedetected includes, for example, physical condition, awakened degree,concentration, fatigue, line-of-sight direction, and the like. Theinside-vehicle environment to be detected includes, for example,temperature, humidity, brightness, smell, and the like. Theinside-vehicle information detection unit 142 supplies data indicatingthe result of detection processing to the situation recognition unit 153of the situation analysis unit 133, the emergency avoidance unit 171 ofthe operation control unit 135, and the like.

The vehicle state detection unit 143 performs detection processing onthe state of the own vehicle on the basis of data or signals from eachunit of the vehicle control system 100. The state of the own vehicle tobe detected includes, for example, speed, acceleration level, steeringangle, presence or absence and details of abnormality, driving operationstate, power seat position and tilt, door lock state, state of othervehicle-mounted devices, and the like. The vehicle state detection unit143 supplies data indicating the result of detection processing to thesituation recognition unit 153 of the situation analysis unit 133, theemergency avoidance unit 171 of the operation control unit 135, and thelike.

The self-position estimation unit 132 performs estimation processing onthe position, orientation, and the like of the own vehicle on the basisof data or signals from each unit of the vehicle control system 100 suchas the outside-vehicle information detection unit 141 and the situationrecognition unit 153 of the situation analysis unit 133. Furthermore,the self-position estimation unit 132 generates a local map to be usedfor self-position estimation as necessary (hereinafter referred to asself-position estimation map). The self-position estimation map is, forexample, a high-precision map using a technique such as simultaneouslocalization and mapping (SLAM). The self-position estimation unit 132supplies data indicating a result of the estimation processing to themap analysis unit 151, the traffic rule recognition unit 152, and thesituation recognition unit 153 of the situation analysis unit 133, andthe like. Furthermore, the self-position estimation unit 132 stores theself-position estimation map in the storage unit 111.

The situation analysis unit 133 performs analysis processing on the ownvehicle and the surrounding situation. The situation analysis unit 133includes the map analysis unit 151, the traffic rule recognition unit152, the situation recognition unit 153, and a situation prediction unit154.

The map analysis unit 151 performs analysis processing on various mapsstored in the storage unit 111, and constructs a map includinginformation necessary for the processing of automated driving whileusing, as necessary, data or signals from each unit of the vehiclecontrol system 100 such as the self-position estimation unit 132, andthe outside-vehicle information detection unit 141. The map analysisunit 151 supplies the constructed map to the traffic rule recognitionunit 152, the situation recognition unit 153, the situation predictionunit 154, and a route planning unit 161, a behavior planning unit 162,and an operation planning unit 163 of the planning unit 134, and thelike.

The traffic rule recognition unit 152 performs recognition processing ontraffic rules around the own vehicle on the basis of data or signalsfrom each unit of the vehicle control system 100 such as theself-position estimation unit 132, the outside-vehicle informationdetection unit 141, and the map analysis unit 151. By the recognitionprocessing, for example, the position and state of traffic lights aroundthe own vehicle, detailed traffic regulation around the own vehicle,lanes that can be traveled, and the like are recognized. The trafficrule recognition unit 152 supplies data indicating a result of therecognition processing to the situation prediction unit 154 and thelike.

The situation recognition unit 153 performs recognition processing onthe situation of the own vehicle on the basis of data or signals fromeach unit of the vehicle control system 100 such as the self-positionestimation unit 132, the outside-vehicle information detection unit 141,the inside-vehicle information detection unit 142, the vehicle statedetection unit 143, and the map analysis unit 151. For example, thesituation recognition unit 153 performs recognition processing on thesituation of the own vehicle, the situation around the own vehicle, thesituation of the driver of the own vehicle, and the like. Furthermore,the situation recognition unit 153 generates, as necessary, a local mapto be used for recognizing the situation around the own vehicle(hereinafter referred to as situation recognition map). The situationrecognition map is, for example, an occupancy grid map.

The situation of the own vehicle to be recognized includes, for example,the position, orientation, movement of the own vehicle (for example,speed, acceleration level, moving direction, and the like), the presenceor absence and details of abnormality, and the like. The situationaround the own vehicle to be recognized includes, for example, the typeand position of a surrounding stationary object, the type, position, andmovement of a surrounding moving object (for example, speed,acceleration level, moving direction, and the like), configuration ofsurrounding roads and road surface conditions, surrounding weather,temperature, humidity, brightness, and the like. The driver state to berecognized includes, for example, physical condition, awakened degree,concentration, fatigue, movement of line-of-sight, driving operation,and the like.

The situation recognition unit 153 supplies data indicating a result ofthe recognition processing (including situation recognition map, asnecessary) to the self-position estimation unit 132, the situationprediction unit 154, and the like. Furthermore, the situationrecognition unit 153 stores the situation recognition map in the storageunit 111.

The situation prediction unit 154 performs situation predictionprocessing regarding the own vehicle on the basis of data or signalsfrom each unit of the vehicle control system 100 such as the mapanalysis unit 151, the traffic rule recognition unit 152, and thesituation recognition unit 153. For example, the situation predictionunit 154 performs prediction processing on the situation of the ownvehicle, the situation around the own vehicle, the situation of thedriver, and the like.

The situation of the own vehicle to be predicted includes, for example,action of the own vehicle, occurrence of abnormality, distance to empty,and the like. The situation around the own vehicle to be predictedincludes, for example, action of a moving object around the own vehicle,change in the state of traffic lights, change in the environment such asthe weather, and the like. The situation of the driver to be predictedincludes, for example, action and physical condition of the driver, andthe like.

The situation prediction unit 154 supplies data indicating a result ofthe prediction processing to the route planning unit 161, the behaviorplanning unit 162, and the operation planning unit 163 of the planningunit 134, and the like together with data from the traffic rulerecognition unit 152 and the situation recognition unit 153.

The route planning unit 161 plans a route to a destination on the basisof data or signals from each unit of the vehicle control system 100 suchas the map analysis unit 151 and the situation prediction unit 154. Forexample, the route planning unit 161 sets a route from the currentposition to the designated destination on the basis of the global map.Furthermore, for example, the route planning unit 161 appropriatelychanges the route on the basis of the situation of traffic congestion,accident, traffic regulation, construction, and the like, physicalcondition of the driver, and the like. The route planning unit 161supplies data indicating the planned route to the behavior planning unit162 and the like.

The behavior planning unit 162 plans the behavior of the own vehicle forsafely traveling the route planned by the route planning unit 161 withinplanned time on the basis of data or signals from each unit of thevehicle control system 100 such as the map analysis unit 151 and thesituation prediction unit 154. For example, the behavior planning unit162 plans start, stop, traveling direction (for example, forward,backward, left turn, right turn, turning, and the like), travel lane,travel speed, passing, and the like. The behavior planning unit 162supplies data indicating the planned behavior of the own vehicle to theoperation planning unit 163 and the like.

The operation planning unit 163 plans the operation of the own vehiclefor implementing the behavior planned by the behavior planning unit 162on the basis of data or signals from each unit of the vehicle controlsystem 100 such as the map analysis unit 151 and the situationprediction unit 154. For example, the operation planning unit 163 plansacceleration, deceleration, travel locus, and the like. The operationplanning unit 163 supplies data indicating the planned operation of theown vehicle to an acceleration-deceleration control unit 172, adirection control unit 173, and the like of the operation control unit135.

The operation control unit 135 controls the operation of the ownvehicle. The operation control unit 135 includes the emergency avoidanceunit 171, the acceleration-deceleration control unit 172, and thedirection control unit 173.

The emergency avoidance unit 171 performs detection processing onemergencies such as collision, scrape, entry into a danger zone, driverabnormality, and vehicle abnormality on the basis of a detection resultby the outside-vehicle information detection unit 141, theinside-vehicle information detection unit 142, and the vehicle statedetection unit 143. In a case where occurrence of an emergency isdetected, the emergency avoidance unit 171 plans the operation of theown vehicle to avoid the emergency such as sudden stop or quick turning.The emergency avoidance unit 171 supplies data indicating the plannedoperation of the own vehicle to the acceleration-deceleration controlunit 172, the direction control unit 173, and the like.

The acceleration-deceleration control unit 172 performsacceleration-deceleration control for implementing the operation of theown vehicle planned by the operation planning unit 163 or the emergencyavoidance unit 171. For example, the acceleration-deceleration controlunit 172 calculates a control target value for the driving forcegeneration device or the braking device for implementing plannedacceleration, deceleration, or sudden stop, and supplies thedrive-affiliated control unit 107 with a control command indicating thecalculated control target value.

The direction control unit 173 performs direction control forimplementing the operation of the own vehicle planned by the operationplanning unit 163 or the emergency avoidance unit 171. For example, thedirection control unit 173 calculates a control target value for asteering mechanism for implementing a travel locus or quick turningplanned by the operation planning unit 163 or the emergency avoidanceunit 171, and supplies the drive-affiliated control unit 107 with acontrol command indicating the calculated control target value.

To perform higher-precision external recognition toward implementationof automated driving and ADAS, many external recognition sensors ofdifferent types are beginning to be installed in vehicles. Meanwhile,each sensor has its strong and weak points. For example, a camera thatcaptures visible light is not good at dark places, and a radar thatdetects reflection of radio waves is not good at objects that do noteasily reflect radio waves, such as persons and animals.

The strong and weak points of each sensor also depend on each detectionprinciple. The strong and weak points of a radar (millimeter waveradar), a camera, and a laser radar (LiDAR) are summarized in Table 1below. In the table, ⊙ means very strong point (having high precision),◯ means strong point (having good precision), and A means weak point(having insufficient precision). However, the detection principle of theradar is to reflect radio waves and measure the distance to an objectand the like, the detection principle of the camera is to capturereflected visible light from a subject, and the detection principle ofthe LiDAR is to reflect light and measure the distance to an object andthe like.

TABLE 1 Sensor type Radar Camera LiDAR Measurement distance ◯ Δ ⊙ Angle,resolution Δ ⊙ ◯ Performance in bad weather ⊙ Δ ◯ Performance at night ⊙◯ ⊙ Classification of object Δ ⊙ ◯

Most of past sensor fusion technologies do not use data detected by asensor whose recognition precision has deteriorated due to the change inthe external environment and the like by supplementing the sensor whoserecognition precision has decreased due to the change in the externalenvironment with another sensor or by switching the combination ofsensors to use.

However, even if the sensor has a low recognition precision, it does notmean that nothing can be recognized from detected data of the sensor.For example, a camera is not good at dark places, but it is possible torecognize to some extent an object in a nearby place or a place wherestreet lights or city lights are illuminating from an image captured bythe camera.

Therefore, the present specification proposes below the technology thatfurther improves the recognition precision by effectively usingdetection signals from sensors with a low recognition precision fromamong a plurality of sensors mounted on vehicles such as a radar, acamera, and a LiDAR.

FIG. 2 schematically shows a functional configuration of an informationprocessing device 1000 to which the technology disclosed in the presentspecification is applied.

The illustrated information processing device 1000 includes a camerarecognition processing unit 1010 that processes a detection signal of acamera 1070, a radar recognition processing unit 1020 that processes adetection signal of a millimeter wave radar 1080, a LiDAR recognitionprocessing unit 1030 that processes a detection signal of a LiDAR 1090,and a fusion processing unit 1040 that performs fusion processing on aprocessing result by each of the recognition processing units 1010 to1030 described above.

External recognition sensors such as the camera 1070, the millimeterwave radar 1080, and the LiDAR 1090 are mounted on the same vehicleafter each installation position is calibrated such that detectionranges become almost the same. Furthermore, it is also assumed thatexternal recognition sensors other than the sensors 1070 to 1090described above are further mounted on the same vehicle. Furthermore, itis also assumed that the camera 1070 includes a plurality of cameras,and that at least part of the cameras is installed to be different fromthe millimeter wave radar 1080 and the LiDAR 1090 in detection range.Outputs of the plurality of cameras may undergo fusion processing by thefusion processing unit 1040.

The camera recognition processing unit 1010 includes a RAW dataprocessing unit 1011 that processes RAW data input from the camera 1070,a signal processing unit 1012 that performs signal processing on the RAWdata, and a recognizer 1013 that recognizes an object from a cameraimage after the signal processing. The RAW data mentioned here is datain which light information captured by an image sensor is recorded as itis. The RAW data processing unit 1011 performs front-end processing(amplification, noise removal, AD conversion, and the like) on the RAWdata, and the signal processing unit 1012 performs back-end processing.The recognizer 1013 may be in either form of hardware that implements apredetermined image recognition algorithm or software that executes therecognition algorithm. The recognizer 1013 outputs information (targetlist) on shape classification of the recognized object (target).Examples of object shape classification recognized by the recognizer1013 include a person, car, sign, sky, building, road, sidewalk, and thelike. The recognizer 1013 may be in either form of hardware thatimplements a predetermined image recognition algorithm or software thatexecutes the recognition algorithm.

As a recognition result by the recognizer 1013, the object shapeclassification (person, car, sign, sky, building, road, sidewalk, andthe like) is output from the camera recognition processing unit 1010 tothe subsequent-stage fusion processing unit 1040. However, therecognizer 1013 does not output a recognition result with lowlikelihood. Therefore, under situations where the recognitionperformance of the recognizer 1013 decreases, such as in bad weather orat night, an information amount output from the recognizer 1013 maydecrease. Furthermore, in the present embodiment, early data beforeobtaining the final recognition result by the recognizer 1013 is alsooutput to the fusion processing unit 1040. The early data mentioned hereincludes a captured image input from the camera 1070 (RAW data), outputdata and data during signal processing from the signal processing unit1012, data during recognition by the recognizer 1013, and the like. Allor part of the early data is output to the fusion processing unit 1040.Because of low likelihood of recognition, it is assumed that the dataduring recognition by the recognizer 1013 includes information regardingan object that is not finally output from the recognizer 1013 (forexample, pedestrian information and the like that is hidden behind anobject such as a car body or a fence and can be recognized onlypartially or fragmentarily). Hereinafter, the early data beforerecognition processing by the recognizer 1013 will be collectivelyreferred to as “RAW data” of the camera 1070 for convenience.

Furthermore, the radar recognition processing unit 1020 includes a RAWdata processing unit 1021 that processes RAW data input from themillimeter wave radar 1080, a signal processing unit 1022 that performssignal processing on the RAW data, and a recognizer 1023 that recognizesan object from a radar detection result after the signal processing. Therecognizer 1023 may be in either form of hardware that implements apredetermined recognition algorithm or software that executes therecognition algorithm. The recognizer 1023 tracks the recognized target(person, car, sign, building, and the like) and outputs a recognitionresult such as a distance, angle of elevation, azimuth, speed, andreflection intensity of each target.

A final recognition result by the recognizer 1023 is output from theradar recognition processing unit 1020 to the subsequent-stage fusionprocessing unit 1040. However, the recognizer 1013 does not output arecognition result with low likelihood. Therefore, it is assumed thatinformation output from the recognizer 1023 regarding objects with weakradar reflection intensity such as a nonmetal will decrease.Furthermore, in the present embodiment, early data before obtaining thefinal recognition result by the recognizer 1023 is also output to thefusion processing unit 1040. The early data mentioned here includes acaptured image input from the millimeter wave radar 1080 (RAW data),output data and data during signal processing from the signal processingunit 1022, data during recognition by the recognizer 1023, and the like.All or part of the early data is output to the fusion processing unit1040. Because of low likelihood of recognition, it is assumed that thedata during recognition by the recognizer 1023 includes information thatis not finally output from the recognizer 1023 (for example, amotorcycle and the like whose reflection intensity is weakened byinfluence of reflected radio waves from a nearby object such as fence orsignboard). Hereinafter, the early data before recognition processing bythe recognizer 1023 will be collectively referred to as “RAW data” ofthe millimeter wave radar 1080 for convenience.

Furthermore, the LiDAR recognition processing unit 1030 includes a RAWdata processing unit 1031 that processes RAW data input from the LiDAR1090, a signal processing unit 1032 that performs signal processing onthe RAW data, and a recognizer 1033 that recognizes an object from aLiDAR detection result after the signal processing. The recognizer 1033may be in either form of hardware that implements a predeterminedrecognition algorithm or software that executes the recognitionalgorithm. The recognizer 1033 tracks the recognized target (person,car, sign, building, and the like) and outputs a recognition result suchas a distance, angle of elevation, azimuth, height, and reflectivity ofeach target.

A final recognition result by the recognizer 1033 is output from theLiDAR recognition processing unit 1030 to the subsequent-stage fusionprocessing unit 1040. However, the recognizer 1013 does not output arecognition result with low likelihood. Furthermore, in the presentembodiment, early data before obtaining the final recognition result bythe recognizer 1033 is also output to the fusion processing unit 1040.The early data mentioned here includes a captured image input from theLiDAR 1090 (RAW data), output data and data during signal processingfrom the signal processing unit 1032, data during recognition by therecognizer 1033, and the like. All or part of the early data is outputto the fusion processing unit 1040. Because of low likelihood ofrecognition, it is assumed that the data during recognition by therecognizer 1033 includes information that is not finally output from therecognizer 1033 (pedestrian information that can be recognized onlypartially or fragmentarily) and the like. Hereinafter, the early databefore recognition processing by the recognizer 1033 will becollectively referred to as “RAW data” of the LiDAR 1090 forconvenience.

FIG. 2 schematically depicts the configuration of each of therecognition processing units 1010 to 1030 for convenience. It is to beunderstood that the detailed internal configuration is determineddepending on the type of sensor, the model and design specification ofthe sensor, and the like. For example, configurations are also assumedin which some or all components of the camera recognition processingunit 1010 are mounted in the unit of the camera 1070, some or allcomponents of the radar recognition processing unit 1020 are mounted inthe unit of the millimeter wave radar 1080, or some or all components ofthe LiDAR recognition processing unit 1030 are mounted in the unit ofthe LiDAR 1090.

Furthermore, in a case where an external recognition sensor (not shown)other than the camera 1070, the millimeter wave radar 1080, and theLiDAR 1090 are mounted on the same vehicle, the information processingdevice 1000 may further include a recognition processing unit includinga RAW data processing unit, a signal processing unit, and a recognizerfor recognition processing based on a detection signal of the sensor. Inthese cases as well, the final recognition result is output from eachrecognition processing unit to the subsequent-stage fusion processingunit 1040, and the RAW data of the sensor is output.

Furthermore, although there is a tendency for a plurality of externalrecognition sensors to be mounted on a vehicle toward implementation ofautomated driving and ADAS (as described above), of course, it is alsoassumed that only one external recognition sensor among the camera 1070,the millimeter wave radar 1080, and the LiDAR 1090 is mounted on onevehicle. For example, it is also assumed that the millimeter wave radar1080 is not used in a case where sufficient external recognitionperformance can be obtained with only the LiDAR 1090, and it is assumedthat a video captured by the camera 1070 is used only for viewing andnot for external recognition. In such a case, it is to be understoodthat the information processing device 1000 is equipped with only thefunctional module corresponding to the sensor to use among therecognition processing units 1010 to 1030 described above, or that allthe functional modules corresponding to each sensor are equipped, butonly the corresponding functional module (or functional module having aninput signal from the sensor) operates and sends an output to thesubsequent-stage fusion processing unit 1040.

The fusion processing unit 1040 performs fusion processing on therecognition result based on each sensor of the camera 1070, themillimeter wave radar 1080, and the LiDAR 1090 mounted on the samevehicle to perform external recognition. In a case where still anotherexternal recognition sensor (not shown) is mounted on the same vehicle,the fusion processing unit 1040 also performs further fusion processingon a detection signal from the sensor. In the present embodiment, thefusion processing unit 1040 performs fusion processing not only on therecognition result of each sensor but also on the RAW data beforerecognition to perform external recognition. Then, the fusion processingunit 1040 outputs an external recognition result obtained by performingthe fusion processing to the vehicle control system.

In the example shown in FIG. 2, the vehicle control system includes anelectronic control unit (ECU) 1050 and an actuator (hereinafter referredto as “ACT”) 1060 that moves the vehicle. The ECU 1050 makesdetermination for automated driving or driving assistance, for example,adaptive cruise control (ACC), lane departure warning (LDW), lanekeeping assist (LKA), autonomous emergency braking (AEB), and blind spotdetection (BSD) on the basis of the external recognition result by thefusion processing unit 1040. Then, the ACT 1060 performs drive controlon each drive unit, that is, operation of the vehicle, such as activecornering light (ACL), brake actuator (BRK), and steering gear (STR), inaccordance with an instruction from the ECU 1050. For example, in a casewhere a road lane is recognized by the fusion processing unit 1040, thevehicle control system controls the travel of the vehicle to prevent thevehicle from deviating from the lane. Furthermore, in a case where anobstacle such as a surrounding vehicle, pedestrian, roadside fence orsignboard is recognized by the fusion processing unit 1040, the vehiclecontrol system controls the travel of the vehicle to allow the vehicleto avoid a collision with the obstacle. Automated driving generallyincludes three steps: “cognition->determination->operation”. Thecognitive step recognizes that there is some object, and thedetermination step determines what is recognized to determine a routeplan of the vehicle. In the configuration example shown in FIG. 2,processing of the cognitive step is mainly performed in the informationprocessing device 1000, processing of the determination step is mainlyperformed by the ECU 1050 in the vehicle control system, and processingof the operation step is mainly performed by the ACT 1060. However,distinction between the cognitive step and the determination step is notstrict, and a part of the cognitive step described in the presentembodiment may be positioned as the determination step. Furthermore, inthe future, design is also expected in which some or all of functions ofprocessing the cognitive step will be the mounted in each sensor unitsuch as the camera 1070, the millimeter wave radar 1080, and the LiDAR1090.

In the present embodiment, the fusion processing unit 1040 includes alate fusion processing unit 1041, an early fusion processing unit 1042,and a hybrid fusion processing unit 1043. The late fusion processingunit 1041 performs fusion processing on the final output (late data) byeach of the recognition processing units 1010 to 1030, that is, on therecognition result by each of the recognizers 1013, 1023, and 1033 toperform external recognition. Furthermore, the early fusion processingunit 1042 performs fusion processing on early data before therecognition distance by each of the recognition processing units 1010 to1030, that is, on RAW data of each sensor of the camera 1070, themillimeter wave radar 1080, and the LiDAR 1090 (as described above) toperform external recognition. Furthermore, the hybrid fusion processingunit 1043 performs fusion processing on either one or more of the finaloutput (late data) by each of the recognition processing units 1010 to1030 and RAW data of either one or more of the recognition processingunits 1010 to 1030 to perform external recognition. Even if likelihoodof the final recognition result by the recognizer of a sensor is low,the hybrid fusion processing unit 1043 has an effect of enhancing therecognition performance by the fusion processing with RAW data ofanother sensor or the same sensor. Then, the fusion processing unit 1040further performs fusion processing on the recognition results of thelate fusion processing unit 1041, the early fusion processing unit 1042,and the hybrid fusion processing unit 1043, or selectively selects therecognition result of the fusion processing units 1041 to 1043, andoutputs the processing result to the subsequent-stage ECU 1050.

The late fusion processing unit 1041, which obtains the finalrecognition results by the recognizers 1013, 1023, and 1033 from therecognition processing units 1010 to 1030, respectively, processesinformation with high reliability of the recognition results. However,there is a problem that the information amount that can be used is smallbecause only the recognition results with high likelihood is output fromthe recognizers 1013, 1023, and 1033 and only the information with highreliability can be obtained.

Meanwhile, since the early fusion processing unit 1042 obtains the RAWdata from the recognition processing units 1010 to 1030 before passingthrough the recognizers 1013, 1023, and 1033, respectively, theinformation amount to be input is very large. However, the RAW data ordata close to the RAW data contains noise. For example, when the camera1070 captures an image of a dark place at night, various informationitems other than an object is included, increasing the possibility oferroneous detection and decreasing the reliability of the information.Furthermore, since the information amount is large, the processingamount is also large.

For example, it is assumed that when the camera 1070 captures an imageof a dark place at night, the captured image (that is, RAW data)contains images of a plurality of persons, as shown in FIG. 3. Since apedestrian 301, who is reflected by street lights and the like andbrightly projected, can be recognized at a high recognition rate, therecognizer 1013 outputs a result of recognizing the pedestrian 301 as atarget. Meanwhile, pedestrians 302 and 302, who are darkly projectedwithout being exposed to street lights and the like, have a lowrecognition rate, and the recognizer 1013 outputs a result of notrecognizing such pedestrians 302 and 302 as a target. That is, althoughthe plurality of pedestrians 301 to 303 is projected in the originalcamera image (that is, RAW data), only information with high reliabilityis output by passing through the recognizer 1013, and the informationamount is narrowed down. Therefore, only information on the pedestrian301 with high reliability is input into the late fusion processing unit1041, and unnecessary information such as noise is omitted, butinformation on the pedestrians 302 and 303 with low reliability is alsoomitted. Meanwhile, information on all the pedestrians 301 to 303 isinput into the early fusion processing unit 1042, and variousinformation items such as noise is also input.

Furthermore, the RAW data of a detection result of the millimeter waveradar 1080 includes intensity distribution of reflected radio wave at aposition where each reflecting object exists (direction and distance) ina predetermined detection range forward of a receiving unit of thereflected radio wave. The RAW data contains intensity data of reflectedradio waves from various objects. However, when passing through therecognizer 1033, the intensity data with low reflection intensity ofradio wave is omitted, only numerical information such as the direction,distance (including depth and width), and speed of the recognized targetis extracted, and the information amount is narrowed down.

For example, as shown in FIG. 4, in a scene where a pedestrian 403 iswalking between two vehicles 401 and 402, it is desirable to be able todetect the pedestrian 403 as well as the vehicles 401 and 402. However,in a case where the millimeter wave radar 1080 is used, the reflectionintensity of an object sandwiched between strong reflectors tends to beweakened. Therefore, when the scene as shown in FIG. 4 is measured withthe millimeter wave radar 1080, as shown in FIG. 5, RAW data 500contains strong reflected waves 501 and 502 from the vehicles 401 and402, respectively, and a weak reflected wave 503 from the pedestrian403. If the detection result of such a millimeter wave radar 1080 passesthrough the recognizer 1033, the reflected wave 503 with low reliabilityas information is omitted, and only the reflected waves 501 and 502 withhigh reliability are recognized as targets, and thus the informationamount is narrowed down. Therefore, only information on the vehicles 401and 402 with high reliability is input into the late fusion processingunit 1041, and unnecessary information such as noise is omitted.However, information on the pedestrian 403 with low reliability is alsoomitted. Meanwhile, information on the pedestrian 403 as well as thevehicles 401 and 402 is input into the early fusion processing unit1042, but various information items such as noise is also input.

In short, the result obtained by performing fusion processing on therecognition result of each of the recognizers 1013, 1023, and 1033 bythe late fusion processing unit 1041 is narrowed down to informationwith high reliability. Therefore, there is a possibility thatinformation with low reliability but high importance may be omitted.Meanwhile, the result obtained by performing fusion processing on theRAW data of each of the recognition processing units 1010 to 1030 by theearly fusion processing unit 1042 has a large amount of information, butthere is a possibility that information with low reliability such asnoise may be taken in.

Therefore, the information processing device 1000 according to thepresent embodiment is configured to obtain an external recognitionresult having a sufficient information amount and high reliability bysupplementing the processing result of the late fusion processing unit1041 having high reliability but narrowed down information amount withthe processing result of the early fusion processing unit 1042 having alarge information amount but also noise. Furthermore, the hybrid fusionprocessing unit 1043 performs fusion processing on either one or more ofthe final outputs (late data) of each of the recognition processingunits 1010 to 1030 and the RAW data of either one or more of therecognition processing units 1010 to 1030. Even if the final recognitionresult by the recognizer of one sensor has low likelihood, the hybridfusion processing unit 1043 can improve the recognition performance byfusion processing with the RAW data of another sensor or the samesensor. That is, the information processing device 1000 is configured torestore important information that will be omitted by the recognizers1013, 1023, or 1033 on the basis of the result of fusion processing onthe RAW data of each external recognition sensor such as the camera1070, the millimeter wave radar 1080, and the LiDAR 1090 by the earlyfusion processing unit 1042.

Note that at the technical level at the time of the present application,the precision of external recognition by the LiDAR 1090 is significantlyhigher than the precision of a camera or millimeter wave radar. However,the recognition result by the LiDAR recognition processing unit 1030 maybe subjected to fusion processing by the late fusion processing unit1041 together with the recognition result of another recognitionprocessing unit 1010 or 1020, and the recognition result by the LiDARrecognition processing unit 1030 may be supplemented with the processingresult of the RAW data by the early fusion processing unit 1042.Furthermore, the LiDAR recognition processing unit 1030 may be subjectedto fusion processing by the hybrid fusion processing unit 1043 togetherwith the recognition result of another recognition processing unit 1010or 1020. In addition, in a case where only the recognition result by therecognizer 1033 of the LiDAR recognition processing unit 1030 issufficient, the late fusion processing unit 1041 does not have toperform fusion processing on the recognition results of the recognizers1013 and 1023 of the camera recognition processing unit 1010 and theradar recognition processing unit 1020, respectively, or the RAW data.

Meanwhile, at the technical level at the time of the presentapplication, the LiDAR 1090 is extremely more expensive than otherexternal recognition sensors such as the camera 1070 and the millimeterwave radar 1080. Therefore, without using the LiDAR 1090 (in otherwords, without mounting the LiDAR 1090 on the vehicle), the informationprocessing device 1000 may be configured to supplement the results offusion processing of the recognition result by the recognizers 1013 and1023 of the camera recognition processing unit 1010 and the radarrecognition processing unit 1020, respectively, by the late fusionprocessing unit 1040, with the result of fusion processing of the RAWdata of each of the camera 1070 and the millimeter wave radar 1080 bythe early fusion processing unit 1042.

Furthermore, from the nature of the LiDAR 1090 using reflected waves oflight, there is a concern that the reliability will deteriorate in theweather that blocks light such as rainfall, snowfall, and fog, and indark places such as at night and in tunnels. Furthermore, similarconcerns apply to the camera 1070. Meanwhile, the reliability of themillimeter wave radar 1080 is not so dependent on the weather and isrelatively stable. Therefore, the information processing device 1000 mayadjust the specific gravity when the fusion processing unit 1040performs fusion processing on the information from each of the sensors1070 to 1090 on the basis of environmental information such as theweather and other external information. For example, in fine weather,the late fusion processing unit 1041 and the early fusion processingunit 1042 in the fusion processing unit 1040 use the recognition resultby the recognizer 1033 of the LiDAR recognition processing unit 1030 andthe RAW data of the LiDAR 1090 with high importance. In a case wherethere is rainfall, snowfall, or fog, or in dark places such as at nightor in tunnels, fusion processing is performed by using with lowimportance or not using the recognition result by the recognizer 1033 ofthe LiDAR recognition processing unit 1030 and the RAW data of the LiDAR1090.

FIG. 6 shows an internal configuration example of the radar recognitionprocessing unit 1020. The radar recognition processing unit 1020includes the RAW data processing unit 1021, the signal processing unit1022, and the recognizer 1023.

The RAW data processing unit 1021 inputs the RAW data of the millimeterwave radar 1080 and performs processing such as amplification, noiseremoval, and AD conversion. The RAW data or data after either ofamplification, noise removal, or AD conversion is output from the RAWdata processing unit 1021 to the early fusion processing unit 1042.

In the example shown in FIG. 6, the signal processing unit 1022 includesa distance detection unit 601 that detects the distance to each targetcaptured by radar, a speed detection unit 602 that detects the speed atwhich each target moves, an angle detection unit 603 that detects theorientation of each target, a tracking unit 604 that tracks the target,and a MISC processing unit 605 that performs other processing. Analgorithm for detecting the distance, orientation, size, and speed ofthe target from the RAW data of the millimeter wave radar 1080 is notparticularly limited. For example, an algorithm developed by themanufacturer of the millimeter wave radar 1080 and the like may beapplied as it is.

When all the processing of respective units 601 to 605 is completed inthe signal processing unit 1022, target information with the distance,orientation, size, speed, and the like detected by radar is output tothe subsequent-stage recognizer 1023. Furthermore, target information ofwhich the distance, orientation, size, and speed cannot be detectedaccurately is not output to the recognizer 1023 as unrecognizable and isomitted. Furthermore, a signal processed by at least one functionalmodule of respective units 601 to 605 is also output to the early fusionprocessing unit 1042.

Note that order in which respective units 601 to 605 perform processingon input data from the RAW data processing unit 1021 is not necessarilyfixed, and it is assumed that the order will be changed as appropriateaccording to product design specifications and the like. Furthermore,not all of the functional modules 601 to 605 described above areessential for the detection signal of the millimeter wave radar 1080. Itis also assumed that the functional modules 601 to 605 are selectedaccording to the product design specifications and the like, or thesignal processing unit 1022 will be equipped with a functional moduleother than the illustrated functional modules.

The recognizer 1023 performs external recognition processing on thebasis of the signal after processing by the signal processing unit 1022according to the predetermined recognition algorithm.

For example, in a scene where an image captured by the vehicle-mountedcamera 1070 is the street as shown in FIG. 7, data before processing bythe recognizer 1023 of the radar recognition processing unit 1020 isshown in FIG. 8, and data after recognition processing by the recognizer1023 is shown in FIG. 9. However, FIG. 8 is an image of the RAW data ofthe millimeter wave radar 1080 or data during processing by the signalprocessing unit 1022. Furthermore, FIG. 9 shows the recognition resultby the recognizer 1023 of the radar recognition processing unit 1020with black blocks. For comparison, FIG. 9 also shows the recognitionresult by the recognizer 1033 of the LiDAR recognition processing unit1030 with gray blocks.

In the scene shown in FIG. 7, it is preferable that a motorcycle 701traveling on a road forward of the vehicle can be recognized as anobstacle. However, houses and fences 702 and 703 are lined up on bothsides of the road (or motorcycle 701). As shown in FIG. 8, the RAW dataof the millimeter wave radar 1080 before processing by the recognizer1023 contains various information items. Although the RAW data of themillimeter wave radar 1080 has side lobes, a strong reflection 801 fromthe motorcycle 701 can be confirmed. Note that relatively weakreflections 802 and 803 from the left and right fences and the like canalso be confirmed. The millimeter wave radar 1080 has high sensitivityto metal, meanwhile has low sensitivity to nonmetal such as concrete.These objects with weak reflection intensity cannot be recognized afterpassing through the recognizer 1023, but the existence can be confirmedfrom the RAW data. Furthermore, with reference to the recognition resultby the recognizer 1023 of the radar recognition processing unit 1020shown in FIG. 9, together with an object 901 that seems to correspond tothe motorcycle 701 near 20 meters ahead, objects 902 and 903 that seemto correspond to the houses and fences 702 and 703 are also recognizedon both sides of the road (or motorcycle 701). In particular, in the RAWdata, the recognized object 901 and the recognized object 902 overlapeach other, but since the data has not undergone recognition processing,even if reflection intensity is weak, the recognized object 901 and therecognized object 902 are recognized in the data. Therefore, the fusionof the RAW data and the recognition result of the recognizer 1023 makesit possible to recognize the recognized object 901 and the recognizedobject 902 as separate objects. Of course, there are many scenes inwhich the motorcycle can be recognized only by the recognizer 1023 ofthe millimeter wave radar 1080. However, the reflection intensity of themotorcycle is weaker than the reflection intensity of the vehicle. Asshown in FIG. 7, if there are other reflectors near the motorcycle, itwill be difficult to capture the motorcycle with only the millimeterwave radar 1080. In data after recognition processing for recognizingfruits with a certain reflection intensity or higher, the recognizedobject 901 and the recognized object 902 are output as one block ofdata. FIG. 21 schematically shows how a motorcycle 2102 is approaching awall 2101 within a detection range 2100 of the millimeter wave radar1080. FIG. 22 schematically shows a result of recognizing the reflectedwave of the millimeter wave radar 1080 obtained from the detection range2100 with the recognizer 1023. With the recognizer 1023, the reflectionintensity less than a predetermined value is omitted, and the reflectionintensity equal to or greater than the predetermined value is recognizedas an object. Therefore, in the example shown in FIG. 22, one lump 2201in which the wall 2101 and the motorcycle 2102 are integrated isrecognized as an object. In contrast, from the RAW data of themillimeter wave radar 1080, even a weak reflection intensity as will beomitted by the recognizer 1023 can be recognized. Therefore, as shown inFIG. 23, it is possible to recognize the reflection from the wall 2101and the reflection from the motorcycle 2102 as separate objects 2301 and2302.

Furthermore, in a scene where an image captured by the vehicle-mountedcamera 1070 is the street as shown in FIG. 10, data before processing bythe recognizer 1023 of the radar recognition processing unit 1020 isshown in FIG. 11, and data after recognition processing by therecognizer 1023 is shown in FIG. 12. However, FIG. 11 is an image of theRAW data of the millimeter wave radar 1080 or data during processing bythe signal processing unit 1022. Furthermore, FIG. 12 shows therecognition result by the recognizer 1023 of the radar recognitionprocessing unit 1020 with black blocks. For comparison, FIG. 12 showsthe recognition result by the recognizer 1033 of the LiDAR recognitionprocessing unit 1030 with gray blocks together.

FIG. 10 is a scene of traveling in a narrow alley sandwiched betweenfences 1001 and 1002 on both sides, and it is preferable that the fences1001 and 1002 on both sides can be recognized as obstacles. As shown inFIG. 11, the RAW data of the millimeter wave radar 1080 beforeprocessing by the recognizer 1023 contains various information items.The fences 1001 and 1002 themselves are not metal and are difficult tobe captured by the millimeter wave radar 1080, but reflections 1101 and1102 that are thought to be caused by cracks or steps in the fences 1001and 1002 can be confirmed. Furthermore, with reference to therecognition result by the recognizer 1023 of the radar recognitionprocessing unit 1020 shown in FIG. 12, the recognizer 1023 candiscretely recognize only some parts 1201 to 1204 where there arereflections from cracks or steps scattered on respective fences 1001 and1002. However, it is difficult to recognize the fences 1001 and 1002 asa whole, and it is difficult to recognize the fences 1001 and 1002 as awhole without using the recognition result of the recognizer 1033 of theLiDAR 1090 (that is, fusion processing). Meanwhile, from the RAW datashown in FIG. 11, even if the reflected waves are weak, the reflectioninformation 1101 and 1102 indicating the existence of the fences can beacquired.

FIG. 13 shows an example of target recognition results when the latefusion processing unit 1041 performs fusion processing on therecognition results of the camera recognition processing unit 1010 andthe radar recognition processing unit 1020 in the information processingdevice 1000. However, “◯” is entered in the recognition result that thetarget is recognized, and “X” is entered in the recognition result thatthe target is not recognized. In a case where the same target issubjected to recognition processing by each of the camera recognitionprocessing unit 1010 and the radar recognition processing unit 1020,four patterns are assumed: in a case where both processing units canrecognize the target (pattern 1), in a case where only one processingunit can recognize the target (patterns 2 and 3), or in a case whereneither can recognize the target (pattern 4). The late fusion processingunit 1041 outputs the target that can be recognized by both the camerarecognition processing unit 1010 and the radar recognition processingunit 1020 as recognizable (in FIG. 13, “◯” is entered). Meanwhile, thetarget that can be recognized by only one of the camera recognitionprocessing unit 1010 and the radar recognition processing unit 1020, andthe target that can be recognized by neither are output asunrecognizable (in FIG. 13, “X” is entered).

Meanwhile, FIG. 14 shows an example of target recognition results whenthe early fusion processing unit 1042 performs fusion processing on theRAW data of the camera 1070 and the millimeter wave radar 1080 in theinformation processing device 1000. However, in respective patterns 1 to4 in FIG. 14, recognition of the same target as in the correspondingpatterns in FIG. 13 is attempted. Furthermore, “◯” is entered in therecognition result that the target is recognized, and “X” is entered inthe recognition result that the target is not recognized. There is alsoan object that is omitted at a determination threshold for therecognizer 113 or 123 by late fusion processing, but that can berecognized by early fusion processing using RAW data before omission atthe determination threshold. However, it should be noted that an objectwith different recognition results between late fusion processing andearly fusion processing has low likelihood of being an actual object.

In pattern 1 where the target can be recognized by both the recognizer1013 of the camera recognition processing unit 1010 and the recognizer1023 of the radar recognition processing unit 1020, the target can besimilarly recognized with the RAW data of the camera 1070 and the RAWdata of the millimeter wave radar 1080. Therefore, the early fusionprocessing unit 1042 outputs that the target can be recognized (in FIG.14, “◯” is entered). That is, in a case where there is no differencebetween the recognition result by the recognizers 1013 and 1023 and therecognition result by the RAW data, the early fusion processing unit1042 outputs the recognition result similar to the recognition result ofthe late fusion processing unit 1041.

Furthermore, in pattern 2 where the target can be recognized by therecognizer 1013 of the camera recognition processing unit 1010 butcannot be recognized by the recognizer 1023 of the radar recognitionprocessing unit 1020, in a case where the target can be recognized onthe basis of the RAW data of the millimeter wave radar 1080, the earlyfusion processing unit 1042 outputs that the target can be recognized.For example, in a case where the reflection intensity is weak and thetarget that is omitted by the recognizer 1023 can be recognized on thebasis of the RAW data, and the like. Therefore, even a target thatcannot be recognized by the late fusion processing unit 1041 can berecognized by using the early fusion processing unit 1042 (see FIG. 15).It can be said that the recognition rate of the target is improved byearly fusion processing using RAW data with abundant information amount.

Furthermore, in pattern 3 where the target cannot be recognized by therecognizer 1013 of the camera recognition processing unit 1010 but canbe recognized by the recognizer 1023 of the radar recognition processingunit 1020, in a case where the target still cannot be recognized fromthe RAW data of the camera 1070, the early fusion processing unit 1042outputs that the target cannot be recognized. That is, in pattern 3, theearly fusion processing unit 1042 outputs the recognition result similarto the recognition result of the late fusion processing unit 1041.

Furthermore, in pattern 4 where the target cannot be recognized by therecognizer 1013 of the camera recognition processing unit 1010 and therecognizer 1023 of the radar recognition processing unit 1020, in a casewhere the target still cannot be recognized from the RAW data of thecamera 1070 but the target can be recognized on the basis of the RAWdata of the millimeter wave radar 1080, the early fusion processing unit1042 outputs a result that there is a possibility that the targetexists. For example, in a case where the reflection intensity is weakand the target that is omitted by the recognizer 1023 can be recognizedon the basis of the RAW data, and the like. Therefore, even a targetthat cannot be recognized by the late fusion processing unit 1041 can berecognized by using the early fusion processing unit 1042 (see FIG. 16).It can be said that the recognition rate of the target is improved byearly fusion processing using RAW data with abundant information amount.However, even by early fusion processing, the recognition rate is nothigh enough, and thus “Δ” instead of “◯” is entered.

Therefore, in the example shown in FIG. 14, in each of pattern 2 andpattern 4, it can be said that the recognition rate of the target isimproved by supplementing the processing result of the late fusionprocessing unit 1041 using the final recognition result that has highreliability but narrows down the information amount on the basis of theresult processed by the early fusion processing unit 1042 using the RAWdata with a large amount of information but also noise

However, as in the case of pattern 1 and pattern 3 in FIG. 14, even ifthe early fusion processing unit 1042 is used, the recognition result bythe late fusion processing unit 1041 does not change in some cases. Ifthe early fusion processing unit 1042 is always operating, there is aconcern about adverse effects such as an increase in processing load andan increase in power consumption of the information processing device1000. Therefore, the recognition processing of the early fusionprocessing unit 1042 may be activated only in a case where necessary.

FIG. 17 schematically shows the configuration example of the informationprocessing device 1000 configured to perform early fusion processingadaptively. However, in FIG. 17, the same functional module as shown inFIG. 1 is denoted with the same reference number.

A determination processing unit 1701 in the fusion processing unit 1042determines whether or not the RAW data of the millimeter wave radar 1080is necessary. In a case where the RAW data is necessary, thedetermination processing unit 1701 requests the radar recognitionprocessing unit 1020 to output the RAW data of the millimeter wave radar1080. For example, the determination processing unit 1701 compares therecognition results of the recognizer 1013 of the camera recognitionprocessing unit 1010 and the recognizer 1023 of the radar recognitionprocessing unit 1020. In a case where the recognition results correspondto pattern 2 or pattern 4 in FIG. 13, the determination processing unit1701 determines that the RAW data is also necessary, and requests theradar recognition processing unit 1020 to output the RAW data of themillimeter wave radar 1080. Alternatively, the determination processingunit 1701 inputs environmental information such as weather or otherexternal information. When a phenomenon is detected in which therecognition rate of the recognizers 1013 and the 1033 of the camerarecognition processing unit 1010 and the LiDAR recognition processingunit 1030 decreases (or reliability of recognition deteriorates), suchas rainfall, snowfall, fog, and dark places such as at night or intunnels, the determination processing unit 1701 may request the radarrecognition processing unit 1020 to output the RAW data of themillimeter wave radar 1080.

In response to the request from the determination processing unit 1701,the radar recognition processing unit 1020 outputs the RAW data of themillimeter wave radar 1080. Then, the early fusion processing unit 1042uses the RAW data to perform early fusion processing, or the hybridfusion processing unit 1043 uses the RAW data to perform hybrid fusionprocessing. Then, in addition to the recognition result by the latefusion processing unit 1041, with reference to the recognition result bythe early fusion processing unit 1042 or the hybrid fusion processingunit 1043, the fusion processing unit 1040 outputs the final recognitionresult.

FIG. 18 shows a processing procedure for performing target recognitionin the information processing device 1000 shown in FIG. 17 in aflowchart form. However, here, for the sake of simplicity ofdescription, the processing procedure is limited to a case where theinformation processing device 1000 performs fusion processing of twosensors, the camera 1070 and the millimeter wave radar 1080.

When object detection processing is started, the camera recognitionprocessing unit 1010 performs image processing on the RAW data (capturedimage) of the camera 1070 (step S1801) and outputs the recognitionresult by the recognizer 1013 (step S1802).

Furthermore, the radar recognition processing unit 1020 performs signalprocessing on the RAW data of the millimeter wave radar 1080 (stepS1803). Then, the radar recognition processing unit 1020 checks whetheror not the output request for the RAW data has been received (stepS1804).

The determination processing unit 1701 in the fusion processing unit1042 compares the recognition results of the recognizer 1013 of thecamera recognition processing unit 1010 and the recognizer 1023 of theradar recognition processing unit 1020 to determine whether or not theRAW data of the millimeter wave radar 1080 is necessary. In a case wherethe RAW data is necessary, the determination processing unit 1701requests the radar recognition processing unit 1020 to output the RAWdata of the millimeter wave radar 1080 (as described above).Specifically, under the situation corresponding to pattern 2 in FIG. 13,the determination processing unit 1701 determines that the RAW data ofthe millimeter wave radar 1080 is necessary.

Here, when the output request for the RAW data has not been received (Noin step S1804), the radar recognition processing unit 1020 outputs therecognition result by the recognizer 1023 (step S1805). Furthermore, onreceipt of the output request for the RAW data (Yes in step S1804), theradar recognition processing unit 1020 outputs the RAW data of themillimeter wave radar 1080 (step S1806), and requests thesubsequent-stage fusion processing unit 1040 to perform early fusionprocessing or hybrid fusion processing using the RAW data of themillimeter wave radar 1080 (step S1807).

Then, the fusion processing unit 1040 performs fusion processing on theprocessing details of the camera recognition processing unit 1010 andthe radar recognition processing unit 1020 (step S1808). When therequest for early fusion processing or hybrid fusion processing has beenreceived (Yes in step S1809), the fusion processing unit 1040 performsearly fusion processing by the early fusion processing unit 1041 orhybrid fusion processing by the hybrid fusion processing unit 1043 (stepS1810). On the other hand, when the request for early fusion processingor hybrid fusion processing has not been received (No in step S1809),the fusion processing unit 1040 performs late fusion processing by thelate fusion processing unit 1041 (step S1811).

FIG. 19 schematically shows another configuration example of theinformation processing device 1000 configured to perform early fusionprocessing adaptively. However, in FIG. 19, the same functional moduleas shown in FIG. 1 is denoted with the same reference number.

When the recognizer 1013 cannot recognize the target, or in a case wherethe recognition rate of the target is not sufficient, the camerarecognition processing unit 1010 determines that the RAW data of themillimeter wave radar 1080 is necessary, and requests the radarrecognition processing unit 1020 to output the RAW data of themillimeter wave radar 1080. In response to the request from the camerarecognition processing unit 1010, the radar recognition processing unit1020 outputs the RAW data of the millimeter wave radar 1080. Then, theearly fusion processing unit 1042 uses the RAW data to perform earlyfusion processing, or the hybrid fusion processing unit 1043 uses theRAW data to perform hybrid fusion processing. Then, in addition to therecognition result by the late fusion processing unit 1041, withreference to the recognition result by the early fusion processing unit1042 or the hybrid fusion processing unit 1043, the fusion processingunit 1040 outputs the final recognition result.

FIG. 20 shows a processing procedure for performing target recognitionin the information processing device 1000 shown in FIG. 19 in aflowchart form. However, here, for the sake of simplicity ofdescription, the processing procedure is limited to a case where theinformation processing device 1000 performs fusion processing of twosensors, the camera 1070 and the millimeter wave radar 1080.

When object detection processing is started, the camera recognitionprocessing unit 1010 performs image processing on the RAW data (capturedimage) of the camera 1070 (step S2001). Then, the camera recognitionprocessing unit 1010 checks whether or not the image processing resultis good (step S2002).

Here, when the image processing result is good (Yes in step S2002), thecamera recognition processing unit 1010 outputs the recognition resultby the recognizer 1013 (step S2003). Furthermore, when the imageprocessing result is not good (No in step S2002), the camera recognitionprocessing unit 1010 requests the radar recognition processing unit 1020to output the RAW data of the millimeter wave radar 1080 (step S2004).Specifically, the image processing result is not good under thesituation corresponding to pattern 4 in FIG. 13.

Furthermore, the radar recognition processing unit 1020 performs signalprocessing on the RAW data of the millimeter wave radar 1080 (stepS2005). Then, the radar recognition processing unit 1020 checks whetheror not the output request for the RAW data has been received (stepS2006).

Here, when the output request for the RAW data has not been received (Noin step S2006), the radar recognition processing unit 1020 outputs therecognition result by the recognizer 1023 (step S2007). Furthermore, onreceipt of the output request for the RAW data (Yes in step S2006), theradar recognition processing unit 1020 outputs the RAW data of themillimeter wave radar 1080 (step S2008), and requests thesubsequent-stage fusion processing unit 1040 to perform early fusionprocessing or hybrid fusion processing on the RAW data of the millimeterwave radar 1080 (step S2009).

Then, the fusion processing unit 1040 performs fusion processing on theprocessing details of the camera recognition processing unit 1010 andthe radar recognition processing unit 1020 (step S3010). When therequest for early fusion processing or hybrid fusion processing has beenreceived (Yes in step S3011), the fusion processing unit 1040 performsearly fusion processing by the early fusion processing unit 1041 orhybrid fusion processing by the hybrid fusion processing unit 1043 (stepS3012). On the other hand, when the request for early fusion processingor hybrid fusion processing has not been received (No in step S3011),the fusion processing unit 1040 performs late fusion processing by thelate fusion processing unit 1041 (step S3013).

INDUSTRIAL APPLICABILITY

The technology disclosed in the present specification has been describedin detail above with reference to the specific embodiment. However, itis obvious that those skilled in the art can modify or substitute theembodiment without departing from the spirit of the technology disclosedin the present specification

The present specification has mainly described the embodiment regardingfusion of vehicle-mounted sensors, but the scope of application of thetechnology disclosed in the present specification is not limited tovehicles. The technology disclosed in the present specification can besimilarly applied to various types of mobile devices, for example,unmanned aerial vehicles such as drones, robots that autonomously movein a predetermined work space (home, office, factory, and the like),ships, aircrafts, and the like. Of course, the technology disclosed inthe present specification can also be similarly applied to informationterminals installed in mobile devices and various non-mobile devices.

In short, the technology disclosed in the present specification has beendescribed in the form of illustration, and details of description of thepresent specification should not be interpreted in a limited manner. Todetermine the spirit of the technology disclosed in the presentspecification, the claims should be considered.

Note that the technology disclosed in the present specification can alsohave the following configurations.

(1) An information processing device including:

a recognition unit configured to perform recognition processing on anobject on the basis of a detection signal of a sensor; and

a processing unit configured to perform fusion processing on first databefore the recognition by the recognition unit and another data. Thisinformation processing device has the effect of being able to recognizemore objects by using the first data including information beforeomission at a determination threshold by the recognition unit for fusionprocessing.

(2) The information processing device according to (1) described above,in which

the sensor includes a millimeter wave radar. This information processingdevice has the effect of being able to recognize more objects byperforming fusion processing on a recognition result having a highlikelihood but a small information amount after being recognized by arecognizer for the millimeter wave radar, with abundant RAW data beforeomission at the determination threshold by the recognizer.

(3) The information processing device according to (2) described above,in which

before the recognition, the recognition unit performs processing of eachof distance detection, speed detection, angle detection of the object,and tracking of the object on the basis of the detection signal of thesensor, and

the first data includes at least one of the detection signal, a distancedetection result of the object, a speed detection result, an angledetection result, or a tracking result of the object. This informationprocessing device has the effect of being able to recognize more objectsby performing fusion processing on RAW data of the millimeter wave radarand information obtained in each stage of signal processing of the RAWdata such as a distance, speed, angle, and tracking result of theobject, and a recognition result of the recognizer.

(4) The information processing device according to any one of (1) to (3)described above, further including a second recognition unit configuredto perform recognition processing on the object on the basis of adetection signal of a second sensor,

in which the processing unit performs at least one fusion processing offusion processing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data. This information processing device has theeffect of being able to recognize more objects by performing fusionprocessing on data before recognition by the recognizers of the firstsensor and the second sensor, and fusion processing on data afterrecognition by the recognizers of the first sensor and the secondsensor.

(5) The information processing device according to (4) described above,in which

the second sensor includes at least one of a camera or a LiDAR. Thisinformation processing device has the effect of being able to recognizemore objects by performing fusion processing on the recognition resultsof the millimeter wave radar and the camera or the LiDAR, and performingfusion processing on the RAW data of the millimeter wave radar and thecamera or the LiDAR.

(6) The information processing device according to (4) or (5) describedabove, in which

the processing unit determines a method of using the first data in thefusion processing on the basis of a recognition result of therecognition unit and a recognition result of the second recognitionunit. This information processing device has the effect of being able torecognize more objects by using the first data adaptively for fusionprocessing on the basis of the recognition result of the recognitionunit and the recognition result of the second recognition unit.

(7) The information processing device according to (6) described above,in which

in a case where likelihood of the recognition by the second recognitionunit is high but likelihood of the recognition by the recognition unitis low, the processing unit uses the first data in the fusionprocessing. This information processing device has the effect of beingable to recognize more objects while avoiding unnecessary fusionprocessing by using the first data adaptively in the fusion processing.

(8) The information processing device according to any one of (4) to (7)described above, in which

the processing unit determines a method of using the first data in thefusion processing on the basis of a recognition result by the secondrecognition unit. This information processing device has the effect ofbeing able to recognize more objects while avoiding unnecessary fusionprocessing by using the first data adaptively in the fusion processing.

(9) The information processing device according to (8) described above,in which

the processing unit uses the first data in the fusion processing in acase where likelihood of the recognition by the second recognition unitis low. This information processing device has the effect of being ableto recognize more objects while avoiding unnecessary fusion processingby using the first data adaptively in the fusion processing.

(10) An information processing method including:

-   -   a recognition step of performing recognition processing on an        object on the basis of a detection signal of a sensor; and

a processing step of performing fusion processing on first data beforethe recognition in the recognition step and another data. Thisinformation processing method has the effect of being able to recognizemore objects by performing fusion processing on second data afterrecognition in the recognition step and the first data includinginformation before omission at a determination threshold in therecognition step.

(11) The information processing method according to (10) describedabove, in which

the processing step includes performing at least one fusion processingof fusion processing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data.

(12) A computer program described in a computer-readable format forcausing a computer to function as:

a recognition unit configured to perform recognition processing on anobject on the basis of a detection signal of a sensor; and

a processing unit configured to perform fusion processing on first databefore the recognition by the recognition unit and another data.

(13) The computer program according to (12) described above, in which

the processing unit performs at least one fusion processing of fusionprocessing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data.

(14) A mobile device including:

a moving means;

a sensor;

a recognition unit configured to perform recognition processing on anobject on the basis of a detection signal of the sensor;

a processing unit configured to perform fusion processing on first databefore the recognition by the recognition unit and another data; and

a control unit configured to control the moving means on the basis of aprocessing result of the processing unit. This mobile device has theeffect of being able to recognize more objects and to complete themoving means so as to avoid collision with the object by performingfusion processing on second data after recognition by the recognitionunit and the first data including information before omission at adetermination threshold by the recognition unit.

(15) The mobile device according to (14) described above, in which

the processing unit performs at least one fusion processing of fusionprocessing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data.

REFERENCE SIGNS LIST

-   100 Vehicle control system-   101 Input unit-   102 Data acquisition unit-   103 Communication unit-   104 Inside-vehicle device-   105 Output control unit-   106 Output unit-   107 Drive-affiliated control unit-   108 Drive-affiliated system-   109 Body-affiliated control unit-   110 Body-affiliated system-   111 Storage unit-   112 Automated driving control unit-   121 Communication network-   131 Detection unit-   132 Self-position estimation unit-   133 Situation analysis unit-   134 Planning unit-   135 Operation control unit-   141 Outside-vehicle information detection unit-   142 Inside-vehicle information detection unit-   143 Vehicle state detection unit-   151 Map analysis unit-   152 Traffic rule recognition unit-   153 Situation recognition unit-   154 Situation prediction unit-   161 Route planning unit-   162 Behavior planning unit-   163 Operation planning unit-   171 Emergency avoidance unit-   172 Acceleration-deceleration control unit-   173 Direction control unit-   1000 Information processing device-   1010 Camera recognition processing unit-   1011 RAW data processing unit-   1012 Signal processing unit-   1013 Recognizer-   1020 Radar recognition processing unit-   1021 RAW data processing unit-   1022 Signal processing unit-   1023 Recognizer-   1030 LiDAR recognition processing unit-   1031 RAW data processing unit-   1032 Signal processing unit-   1033 Recognizer-   1040 Fusion processing unit-   1041 Late fusion processing unit-   1042 Early fusion processing unit-   1043 Hybrid fusion processing unit-   1050 ECT-   1060 Actuator (ACT)-   1070 Camera-   1080 Millimeter wave radar-   1090 LiDAR-   601 Distance detection unit-   602 Speed detection unit-   603 Angle detection unit-   604 Tracking unit-   605 MISC processing unit

1. An information processing device comprising: a recognition unitconfigured to perform recognition processing on an object on a basis ofa detection signal of a sensor; and a processing unit configured toperform fusion processing on first data before the recognition by therecognition unit and another data.
 2. The information processing deviceaccording to claim 1, wherein the sensor includes a millimeter waveradar.
 3. The information processing device according to claim 2,wherein before the recognition, the recognition unit performs processingof each of distance detection, speed detection, angle detection of theobject, and tracking of the object on a basis of the detection signal ofthe sensor, and the first data includes at least one of the detectionsignal, a distance detection result of the object, a speed detectionresult, an angle detection result, or a tracking result of the object.4. The information processing device according to claim 1, furthercomprising a second recognition unit configured to perform recognitionprocessing on the object on a basis of a detection signal of a secondsensor, wherein the processing unit performs at least one fusionprocessing of fusion processing on third data before the recognition bythe second recognition unit and the first data, fusion processing onfourth data after the recognition by the second recognition unit and thefirst data, fusion processing on the first data and second data afterthe recognition by the recognition unit, or fusion processing on thefourth data and the second data.
 5. The information processing deviceaccording to claim 4, wherein the second sensor includes at least one ofa camera or a LiDAR.
 6. The information processing device according toclaim 4, wherein the processing unit determines a method of using thefirst data in the fusion processing on a basis of a recognition resultof the recognition unit and a recognition result of the secondrecognition unit.
 7. The information processing device according toclaim 6, wherein in a case where likelihood of the recognition by thesecond recognition unit is high but likelihood of the recognition by therecognition unit is low, the processing unit uses the first data in thefusion processing.
 8. The information processing device according toclaim 4, wherein the processing unit determines a method of using thefirst data in the fusion processing on a basis of a recognition resultby the second recognition unit.
 9. The information processing deviceaccording to claim 8, wherein the processing unit uses the first data inthe fusion processing in a case where likelihood of the recognition bythe second recognition unit is low.
 10. An information processing methodcomprising: a recognition step of performing recognition processing onan object on a basis of a detection signal of a sensor; and a processingstep of performing fusion processing on first data before therecognition in the recognition step and another data.
 11. Theinformation processing method according to claim 10, wherein theprocessing step includes performing at least one fusion processing offusion processing on third data before the recognition by the secondrecognition unit and the first data, fusion processing on fourth dataafter the recognition by the second recognition unit and the first data,fusion processing on the first data and second data after therecognition by the recognition unit, or fusion processing on the fourthdata and the second data.
 12. A computer program described in acomputer-readable format for causing a computer to function as: arecognition unit configured to perform recognition processing on anobject on a basis of a detection signal of a sensor; and a processingunit configured to perform fusion processing on first data before therecognition by the recognition unit and another data.
 13. The computerprogram according to claim 12, wherein the processing unit performs atleast one fusion processing of fusion processing on third data beforethe recognition by the second recognition unit and the first data,fusion processing on fourth data after the recognition by the secondrecognition unit and the first data, fusion processing on the first dataand second data after the recognition by the recognition unit, or fusionprocessing on the fourth data and the second data.
 14. A mobile devicecomprising: a moving means; a sensor; a recognition unit configured toperform recognition processing on an object on a basis of a detectionsignal of the sensor; a processing unit configured to perform fusionprocessing on first data before the recognition by the recognition unitand another data; and a control unit configured to control the movingmeans on a basis of a processing result of the processing unit.
 15. Themobile device according to claim 14, wherein the processing unitperforms at least one fusion processing of fusion processing on thirddata before the recognition by the second recognition unit and the firstdata, fusion processing on fourth data after the recognition by thesecond recognition unit and the first data, fusion processing on thefirst data and second data after the recognition by the recognitionunit, or fusion processing on the fourth data and the second data.